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Abstract 

Consider  the  forward  link  of  a  mobile  communications  system  with  a 
single  transmitter  and  connecting  to  K  destinations  via  randomly  varying 
channels.  Data  arrives  in  some  random  way  and  is  queued  according  to 
the  K  destinations  until  transmitted.  Time  is  divided  into  small  schedul¬ 
ing  intervals.  The  new  generation  of  systems  can  estimate  the  channel 
(e.g,  via  pilot  signals)  and  use  this  information  for  scheduling.  The  issues 
are  the  allocation  of  transmitter  power  and/or  time  and  bandwidth  to 
the  various  queues  in  a  queue  and  channel-state  dependent  way  to  assure 
stability  and  good  operation.  The  decisions  are  made  at  the  beginning 
of  the  scheduling  intervals.  Stochastic  stability  methods  are  used  both 
to  assure  that  the  system  is  stable  and  to  get  appropriate  allocations, 
under  very  weak  conditions.  The  choice  of  Liapunov  function  allows  a 
choice  of  the  effective  performance  criteria.  The  resulting  controls  are 
readily  implementable  and  allow  a  range  of  tradeoffs  between  fairness  and 
queue  lengths.  The  various  extensions  allow  a  large  variety  of  schemes  of 
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it  provided. 
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current  interest  to  be  covered.  The  channel-state  process  might  be  well- 
estimated  or  only  partially  known.  All  essential  factors  are  incorporated 
into  a  “mean  rate”  function,  so  that  the  results  cover  many  different  sys¬ 
tems.  Because  of  the  non-Markovian  nature  of  the  problem,  we  use  the 
perturbed  Stochastic  Liapunov  function  method,  which  is  well  adapted  to 
such  problems.  The  method  is  simple,  effective,  and  new.  A  heavy  traffic 
approximation  when  the  controllable  part  of  the  power  is  small  is  in  [4] . 


1  Introduction 

We  consider  the  problem  of  power  and  time  control  for  the  forward  link  of  a 
mobile  communications  system  when  the  connecting  channels  are  randomly  time 
varying.  The  control  can  be  over  allocated  time,  power,  and  bandwidth,  or  over 
all  simultaneously.  More  discussion  of  the  possibilities  appears  in  Section  4. 
There  are  K  queues  at  the  base  station,  each  receiving  data  according  to  some 
random  process,  and  each  associated  with  a  unique  mobile  destination.  The 
(nonnegative)  content  of  any  queue  is  simply  its  initial  content  plus  its  arrivals 
minus  its  departures  to  date.  Time  is  divided  into  small  scheduling  intervals 
of  length  As  (typically,  in  the  low  tens  of  milliseconds)  and  the  transmitter 
decisions  concerning  the  possible  power,  time,  and  bandwidth  allocations  are 
made  at  the  beginning  of  the  intervals.  With  the  appropriate  use  of  pilot  signals 
the  new  generation  of  systems  has  the  ability  to  estimate  key  properties  of  the 
channel  (e.g.,  S/N  ratios)  and  to  use  this  information  to  determine  the  controls. 
Such  an  approach  would  greatly  improve  the  performance  [1],  More  generally, 
there  might  be  only  partial  knowledge  of  the  channel.  The  actual  physical 
situation  that  corresponds  to  a  channel  state  value  or  to  an  estimate  of  the  value 
is  unimportant.  Each  value  of  the  state  or  of  its  estimate  corresponds  to  some  set 
of  allowed  possibilities  for  the  coding  and  resource  allocations.  For  simplicity,  it 
is  assumed  that  the  controller  quantizes  the  state  of  the  K  channels  into  a  finite 
number  of  values,  indexed  by  (a  A^-vector)  j:  each  value  of  j  denotes  the  state 
of  all  of  the  individual  channels.1  The  individual  components  of  the  channel 
state  might  or  might  not  be  mutually  independent.  The  rates  of  transmission 
are  measured  in  terms  of  packets.  The  queue  state  and  data  arrival  processes 
need  not  be  Markovian.  The  actual  transmission  schemes  that  are  allowed  are 
quite  general.  They  can  be  based  on  TDMA,  CDMA,  bit  interval  control,  or  on 
various  combinations. 

To  best  accommodate  issues  of  fairness  as  well  as  the  particular  performance 
criteria  of  interest,  the  power  and  time  allocation  in  each  scheduling  interval 
should  depend  on  the  lengths  of  the  queues  as  well  as  on  the  current  knowl¬ 
edge  of  the  (vector- valued)  state  of  the  channel.  The  resulting  optimal  control 
problem  is  quite  difficult.  The  dimension  (A)  might  be  high,  the  queue-length 
processes  might  not  be  Markovian  under  any  control  scheme,  and  uncertainty  of 

1The  finiteness  of  the  number  of  channel  states  is  assumed  for  notational  convenience,  so 
that  terms  such  as  the  C'b  n  in  the  proof  of  Theorem  3.1  can  be  defined  and  manipulated  in 
a  simple  way. 
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the  channel  state  can  lead  to  a  complicated  filtering  problem.  These  complica¬ 
tions  suggest  an  approximation  approach.  The  small  duration  of  the  scheduling 
interval  As  and  the  (usually)  fast  rate  of  change  of  the  channel-state  process 
suggest  an  approach  to  the  control  problem  that  is  based  on  a  “crude  mean 
flow”  idea.  Loosely  speaking,  we  require  only  that  there  is  some  allocation  un¬ 
der  which  an  appropriately  defined  “mean”  system  is  stable  (stability  is  defined 
in  the  next  section),  and  then  the  stability  approach  can  be  carried  out  under 
quite  general  conditions  for  a  great  variety  of  physical  systems. 

The  conditions  of  typical  applications  (e.g.,  Rayleigh  fading)  do  not  usually 
allow  a  Markov  representation  of  the  queue-length  and  channel-state  processes, 
even  under  “feedback”  control;  hence  the  usual  methods  of  stability  analysis, 
which  require  a  Markov  model,  are  inappropriate.  Because  of  the  non-Markov 
and  randomly-time-varying  nature  of  the  problem,  the  stability  is  proved  using 
the  perturbed  Liapunov  function  methods  of  [8].  With  this  method,  one  starts 
with  a  basic  Liapunov  function  that  works  for  a  simpler  (e.g.,  a  “crude  mean 
flow”)  system,  and  then  finds  a  suitable  perturbation  that  can  be  applied  to 
the  actual  non-Markov  physical  system.  Analogously  to  the  usual  “stability 
method”  procedure  that  is  used  to  get  controls,  the  controls  are  determined 
by  maximizing  the  (conditional  expectation,  given  the  current  data,  of  the) 
negative  of  the  gradient  of  the  basic  (not  the  perturbed)  Liapunov  function. 
The  development  uses  a  basic  Liapunov  function  that  is  a  polynomial  which  is 
the  sum  of  terms,  each  depending  on  only  one  component  of  the  state  of  the 
queue.  This  seems  to  be  adequate  for  applications  at  this  time.  It  is  seen  that 
more  general  polynomials  can  be  used  without  additional  difficulty.  Indeed,  the 
basic  Liapunov  function  can  be  rather  arbitrary,  the  main  requirement  being 
that  the  gradient  has  positive  components.2 

Stability  is  of  interest  partly  because  it  assures  a  robustness  of  behavior  to 
“small”  changes  in  the  system.  For  this  reason  an  analysis  based  on  Markov 
models  is  inadequate.  No  physical  model  is  truly  Markov.  If  an  approximating 
Markov  model  is  used  for  analysis,  then  it  is  important  to  know  that  the  stability 
holds  for  the  original  systems  that  are  approximated.  The  perturbed  Liapunov 
function  method  allows  such  robustness  to  be  confirmed.  Additionally,  mild 
modifications  allow  the  basic  data  (e.g.,  the  probabilistic  structure  of  the  data 
arrival  process)  to  be  time  varying.  The  Liapunov  function  can  be  chosen  to 
reflect  the  relative  importance  of  the  various  queues  or  even  to  accommodate 
constraints.  For  example,  if  it  is  desired  to  minimize  the  time  that  queue  i  is 
above  a  value  A,;,  one  can  let  the  dependence  of  the  Liapunov  function  on  the 
state  of  queue  i  be  very  large  when  it  exceeds  A;.  Analogously  to  the  treatment 
of  a  related  problem  in  [5],  where  a  sequence  of  systems  scaled  by  speed  was 
considered,  one  can  allow  a  received  packet  to  be  rejected  by  the  receiver  if  it 
decides  that  there  are  too  many  errors.  Then  the  acceptance  probability  will 
also  depend  on  the  allocated  power. 

The  assumptions  and  the  precise  definition  of  stability  are  in  Section  2, 

2For  example,  if  we  wish  to  more  or  less  equalize  the  lengths  of  the  various  queues,  one 
can  add  terms  to  the  Liapunov  function  that  depend  on  the  differences  of  the  values  of  the 
queue  components. 
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where  the  assumptions  are  discussed  and  seen  to  be  unrestrictive,  and  a  simple 
example  given.  The  main  theorem  is  stated  and  proved  in  Section  3.  The  scope 
of  the  approach  and  simplicity  of  application  can  be  seen  from  the  examples 
in  Sections  4  and  5,  which  can  be  read  without  knowledge  of  Section  3.  Some 
canonical  classes  of  physical  models  (and  various  combinations  and  extensions) 
are  discussed  in  Section  4.  These  include  the  standard  CDMA  and  TDMA  and 
show  that  many  types  of  constraints  can  be  handled.  The  approach  and  results 
are  the  same  when  the  channel  is  only  partly  known  or  retransmission  of  poorly 
received  packets  is  called  for.  A  similar  analysis  can  be  used  when  there  are 
multiple  antennas  [15]  and  frequencies.  Multiple  antennas  and  frequencies  are 
used  in  the  space-time  coded  OFDM  (orthogonal  frequency  division  multiplex¬ 
ing)  approach  [10,  13],  part  of  the  purpose  of  which  is  to  provide  “space-time 
diversity,”  to  partially  neutralize  the  effects  of  channel  variations.  But,  via  the 
approach  of  this  paper  and  given  sufficient  information  on  the  channel,  one 
might  sometimes  find  it  preferable  to  use  only  the  most  appropriate  combina¬ 
tions  of  antennas  and  frequencies  for  the  various  sources  at  any  time.  There 
are  no  alternative  approaches  that  can  handle  the  same  breadth  of  applications. 
Section  5  contains  a  comparison  of  the  obtained  controls  with  optimal  controls 
for  a  closely  related  problem. 

It  will  be  seen  that  the  approach  is  quite  natural  and  general  for  the  prob¬ 
lems  of  concern.  For  the  special  case  of  problems  of  the  type  of  Example  4.2 
where  the  rate  of  transmission  is  proportional  to  power,  [1,  11]  obtain  rules  for 
power  allocation  whose  form  is  similar  to  ours  and  which  are  based  on  stability 
considerations,  although  the  method  uses  large  deviations  estimates  and  the 
setup  is  Markovian.  The  reference  [14]  was  perhaps  the  first  to  consider  the 
problem  of  dynamic  power  allocation  when  the  channels  are  time- varying.  But, 
since  their  channel-rate  and  data-arrival  processes  are  all  i.i.cl.  sequences,  the 
range  of  applications  is  very  small.  The  reports  [2,  3]  deal  with  related  prob¬ 
lems.  They  prove  interesting  results  concerning  the  limit  (as  t  — »  oo)  of  (queue 
length  at  t)/t,  and  show  that  (under  appropriate  conditions)  this  limit  is  zero. 
This  is  used  to  show  that  the  integral  of  the  “rates”  of  transmission  per  unit 
time  converges.  But  such  a  result  does  not  imply  stability  of  the  queue  length 
process,  since  it  can  grow  sublinearly.  [Consider,  for  example,  a  process  mod¬ 
eled  by  a  random  walk  in  one  dimension,  with  reflection  at  the  origin,  where 
the  queue  is  only  null  recurrent.]  Their  scheduling  is  done  continuously,  rather 
than  discretely  in  time.  They  allocate  a  single  resource  (e.g.,  bandwidth)  and 
the  rate  is  proportional  to  the  allocation.  This  focus  implies  a  more  restricted 
set  of  applications.  For  example,  it  does  not  seem  to  be  able  to  handle  the 
“discrete  selection”  bit  interval  controls,  the  multiple  antenna  problem,  or  the 
joint  bandwidth-power  allocation  problem,  or  nonlinear  dependencies  on  the 
controls  in  general  (as  in  Example  4.1).  Their  setup  and  proofs  are  much  more 
complicated  that  that  here.  Stability  of  the  “rate  of  transmission”  process  in 
our  case  follows  from  Theorem  3.1.  For  models  that  we  also  cover,  the  decision 
rules  would  be  similar  for  appropriate  selections  of  the  Liapunov  function. 

The  work  that  is  closest  to  ours  is  the  interesting  report  [12],  where  the 
channel-state  process  is  Markov,  the  data  input  sequence  is  i.i.cl.,  and  a  “com- 
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plete  resource  pooling”  condition  is  required.  The  decision  rule  is  the  same  as 
ours  for  a  quadratic  Liapunov  function.  The  emphasis  is  on  stability  in  the 
heavy  traffic  limit,  and  showing  how  the  problem  simplifies  there.  Stability  in 
the  heavy  traffic  limit  does  not  imply  stability  for  any  one  of  the  “prelimit” 
processes.  See  also  [5]  for  a  stability  analysis  as  the  heavy  traffic  regime  is 
approached. 

2  Assumptions  and  the  Control  Rule 


Comment  on  controls.  For  purposes  of  concrete  visualization,  one  could 
suppose  that  the  controls  are  over  either  the  power  and/or  time  or  bandwidth 
allocated  to  each  queue.  Other  forms  are  noted  in  the  examples  in  Section  4.  Let 
Ui(j,x)  denote  the  control  applied  to  queue  i  in  a  scheduling  interval,  when  the 
(vector-valued)  queue  state  is  x  and  the  estimate  of  the  (vector-valued)  channel 
state  is  j.  Some  component  of  the  control  might  also  be  a  probability;  e.g.,  as 
in  the  example  at  the  end  of  the  section,  where  the  control  is  the  probability  of 
selecting  a  queue  in  an  interval.  The  constraints  need  to  be  taken  into  account  in 
defining  admissible  controls.  For  example,  the  total  time  scheduled  in  an  interval 
cannot  be  greater  than  As.  In  the  simplest  model,  where  iq(j,  x)  denotes  power, 
the  total  allocated  power  cannot  be  greater  than  that  available  and  then  we  have 

^2ui(j,x)  <  p,,  (2.1) 

i 

p  is  the  (real-valued)  total  available  power.  More  generally,  one  might  only  re¬ 
quire  that  the  power  “  average  locally  in  time”  to  that  available.  Other  possible 
constraints  might  be  a  maximum  length  for  any  queue,  or  the  control  might 
have  to  take  values  in  some  discrete  set.  Obviously,  the  queue  contents  Xi  must 
be  nonnegative.  Other  examples  are  given  in  Section  4. 

Assumption  (A2.1)  gives  the  general  definition  of  admissibility  and  gener¬ 
alizes  (2.1).  Unless  otherwise  mentioned,  when  several  users  are  transmitting 
simultaneously,  it  is  assumed  for  simplicity  that  the  mutual  interference  is  ig- 
norable.  If  this  is  not  the  case,  then  the  same  method  can  be  used,  but  the  rates 
for  any  particular  channel  will  depend  on  the  vector  of  power  allocations. 

Definitions.  The  n-th  scheduling  interval  is  the  real  time  interval  (nAs,  nAs  + 
A'9],  and  the  decisions  for  it  are  made  at  time  nAs.  Let  Ln  denote  the  value  of 
the  random  channel  state  process  at  time  nAs.  Let  ajjH  (resp.,  dz,n)  denote  the 
number  of  packets  arriving  to  (resp.,  transmitted  from)  queue  i  in  scheduling 
interval  n.  The  amount  sent  from  queue  i  in  any  scheduling  interval  depends 
only  on  the  channel  and  full  queue  states  at  the  beginning  of  the  interval,  and 
the  control  allocations  to  queue  i  in  the  interval.  Let  Tn  denote  the  minimal 
cr-algebra  that  measures  the  data  (channel  states,  initial  queue  states,  arrivals, 
control  values)  up  to  and  including  time  nAs,  and  let  En  denote  the  associated 
conditional  expectation. 
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Stability:  Definition.  Owing  to  the  non-Markovianness,  an  appropriate 
definition  of  stability  is  a  “uniform  mean  recurrence  time”  property,  as  follows. 
Suppose  that  there  are  0  <  qo  <  oo  and  a  real- valued  function  F(-)  >0  such 
that  the  following  holds:  For  any  n,  and  o\  =  minjfc  >  n  :  |a:(fcA,s)|  <  go},  we 
have3 

En  [tii  -  n]  <  F(x(nAs))I{\x{nAa)\>qo}.  (2.2) 

Then  the  system  is  said  to  be  stable.4  The  definition  implies  recurrence  to 
some  compact  set.  If  the  (absolute  value  of  the  state  of)  the  process  reaches  a 
level  qi  >  qo,  then  the  conditional  expectation  of  the  time  required  to  return 
to  a  value  qo  or  smaller  is  bounded  by  a  function  of  q±,  uniformly  in  the  past 
history  and  in  n.  This  implies  that  the  sequence  |:r(nAs)}  is  tight  or  bounded 
in  probability  (see,  for  example,  [8,  Theorem  8,  Chapter  6]). 5 

The  basic  Liapunov  function.  Unless  otherwise  noted,  for  simplicity  in  the 
computations  (and  because  they  yield  controls  that  are  readily  implemented) 
the  basic  Liapunov  functions  will  be  polynomials  of  the  form6 

V(x)  =  y +  hj)Pi,  (2.3) 


where  hi  >  0,  Cj  >  0,  and  pi  >  2.  The  hi  serve  the  purpose  of  reducing  the 
dependence  of  the  controls  on  the  queue  size  if  the  content  is  small.  Define 
p  =  maxip,;.  Let  Vx{-)  (with  components  VXi(-))  denote  the  gradient  of  V(-). 

Assumptions.  The  assumptions  are  discussed  after  being  stated. 

(A2.1)  There  is  a  real  (resp.,  vector7)  valued  function8  /(•)  such  that  the  con¬ 
trols  satisfy 

<  Jl,  all  j,  (2.4) 

where  u(x)  =  {ui(j,x);i,j}  and  Jl  is  a  real  (resp.,  vector)  valued  constant. 
Furthermore,  there  is  a  set  U  such  that  u(x)  G  U  for  each  x.  (This  defines  the 
set  of  admissible  controls.) 

(A2.2)  The  maximum  number  that  can  be  sent  from  queue  i  over  a  scheduling 
interval  is  bounded  by  the  minimum  of  (a  constant  times  As)  and  Xi.  [I.e.,  the 

3The  stability  condition  in  [14]  requires  only  that  return  times  be  finite  w.p.l,  but  the 
mean  return  time  might  still  be  infinite.  The  form  (2.2),  based  on  mean  return  times,  seems 
more  appropriate  in  applications. 

4Restricting  attention  to  the  sampling  times  nAs  is  unrestrictive  in  view  of  (A2.6). 

5 A  sequence  {An}  is  bounded  in  probability  if  limK^.oo  supn  P{|An|  >  ac}  =  0. 

6 Other  polynomial  forms  can  be  used  in  the  development,  provided  that  V(x)  — >  oo  as 
x  — ►  oo,  VXi(x)  —>  0  as  Xi  — >  0,  and  VXi(x)  — >  oo  as  X{  — >  oo,  VXi(x)/V(x)  — >  0  as  x  — >  oo, 
and  Enen/\Vx(x)\  — >  0  as  x  — >  oo,  where  en  is  defined  in  (3.5). 

7We  have  the  vector  case  if,  for  example,  both  power  and  time  are  being  constrained  or  in 
the  multiple  antenna  case. 

8 Since  the  x  and  j  variables  are  discrete,  we  do  not  need  to  be  concerned  with  measurability 
issues. 
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actual  number  of  packets  sent  can  be  no  greater  than  x,;.]  There  are  bounded 
functions  g,(-)  with  values  gi(j,x,Ui(j,x))  such  that  if  x(nAs)  =  x  and  Ln  =  j 
then  the  conditional  (given  the  data  to  nAs)  mean  number  of  packets  sent  from 
queue  i  on  (nAs,nAs  +  As]  can  be  represented  as 


Endian  A  Cji  ,  X,  Ui  (j,  x))  . 


(2.5) 


(A2.3)  There  are  positive  constants  A <  K ,  such  that  the  sums 

OO 

Cln  =  Y,En\ai,i-  ASA“],  i<K, 

l—n 

are  well  defined  and  bounded,  uniformly  in  n. 

(A2.4)  There  are  II/  >  0  such  that  II j  =  1  and  En  [l{Lt=j}  —  Hy]  — >  0  fast 
enough  as  l  —  n  — >  oo  so  that  ErJ{ Lt=j}  ~  II j  |  is  bounded  uniformly  in 

n. 


(A2.5)  Define  the  average  gi(x,u(x))  =  -  Hjgi(J,  x,  Ui(j,  x)).  There  are  cq  > 

0,  Kq  >  0,  and  an  admissible  control  u(x)  =  {ui(j,  x);  i,j},  such  that  if  Xi  >  Kq 
then 

K  -9i{x,u(x))  <  -c0 

and  gi(j,x,Ui(j,x)))  does  not  depend  on  x.  9 
(A2.6)  There  is  a  constant  C  such  that  for  each  i 

En  [dij  <  CAS,  l  <  Pi, 

where  Pi  is  defined  by  (2.3). 

The  control  rule.  Controls  satisfying  (A2.1)  are  called  admissible.  The  actual 
control  is  any  admissible  control  that  maximizes  in 

max  | F.  V-Xj (x)gi(j,x,Uj(j,x)) | .  (2.6) 

It  is  assumed  that  there  is  at  least  one  maximizing  admissible  control  Ui  ( j ,  x) . 

Discussion  of  the  conditions.  If  control  is  over  power  only  and  the  total 
instantaneous  power  must  be  no  greater  than  /Z,  and  can  take  only  some  finite  set 

9The  Kq  is  introduced  in  (A2.5)  only  because  if  the  queue  content  is  smaller  than  the 
maximum  of  what  can  be  transmitted  on  a  scheduling  interval,  then  the  mean  (weighing  with 
the  Ilj)  output  might  be  too  small  to  assure  the  — Co  value.  For  example,  if  a  queue  is  empty, 
then  the  arrivals  will  exceed  the  departures.  Kq  will  be  of  the  order  of  As. 
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of  values,  then  (2.4)  reduces  to  (2.1)  and  U  is  the  set  of  allowed  discrete  values. 
Condition  (A2.4)  would  hold  if  the  channel  state  process  were  a  function  of  a 
finite-state  ergodic  Markov  process.  It  need  not  be  Markovian  itself.  Loosely 
speaking,  it  would  hold  if  the  future  becomes  less  and  less  predictable  (given 
the  distant  past)  and  the  conditional  likelihood  of  any  channel  state  approaches 
an  average  value.  Condition  (A2.5)  is  likely  very  close  to  being  necessary.  It 
concerns  an  averaged  system,  where  the  input  is  just  a  fluid  flow  at  the  mean 
rate  and  the  output  is  also  a  fluid  flow,  whose  value  is  the  mean  over  all  channel 
states  (i.e.,  there  is  no  randomness).  If  there  was  no  control  under  which  the 
condition  held,  then  even  this  mean  system  would  be  unstable. 

Condition  (A2.6)  will  hold  if  the  inputs  occur  continuously  in  time,  but  at  a 
bounded  maximum  rate  (the  usual  case) .  Then  the  maximum  input  to  any  queue 
over  a  scheduling  interval  is  proportional  to  As.  Alternatively,  suppose  that  the 
input  rate  is  unbounded  and  a  batch  of  packets  can  arrive  all  at  once.  Suppose 
(without  loss  of  generality)  that  they  arrive  at  instants  kAs ,  k  =  1,2,...,  and 
have  bounded  pth  moments.  Let  the  values  over  the  scheduling  intervals  be 
mutually  independent,  and  let  the  probability  of  an  input  to  queue  i  during  any 
interval  be  c* A3  (a  discrete  time  Poisson  process).  Then  (A2.6)  holds. 


Comments  on  (A2.3).  The  following  simple  examples  illustrate  (A2.3).  First 
suppose  that  the  {a,;jn;*,n}  are  mutually  independent  with  the  inputs  to  each 
queue  being  identically  distributed,  with  A?  defined  so  that  Eai^n  =  ASA?.  Then 
the  summands  in  C“n  are  all  zero  and  C“ra  =  0.  If  the  input  sequence  is  only 
M -dependent,  then  all  but  at  most  the  first  M  summands  in  Cfn  are  zero. 

Let  us  now  examine  Cfn  more  closely  to  understand  why  our  requirement  on 
its  convergence  and  boundedness  is  reasonable.  Henceforth  let  us  lump  together 
all  arrivals  in  the  nth  interval  and  suppose  (w.l.o.g)  that  it  occurs  at  the  end. 
Consider  queue  i.  First  suppose  that  arrivals  occur  in  batches  at  (real  time) 
moments  r“n,  i  =  1,2,...,  which  is  assumed  to  be  a  renewal  process  with  values 
in  {nAs,  n  =  1,2,...}.  Define  the  mean  interval  m“  =  £[rfn+1  —  r“n],  and  let 
the  number  of  arrivals  at  the  renewal  time  r“n  be  denoted  by  u“n.  For  each 
queue  i,  let  the  vfn,  n  <  oo,  be  mutually  independent  and  identically  distributed 
with  mean  vf,  and  suppose  that  the  sequences  for  the  different  queues  are 
mutually  independent.  Define  A?  =  vf/mf  and  let  As /z? 'n  j,  Z  =  1,2,...,  denote 
the  sequence  of  renewal  times  that  occur  after  time  nAs.  By  the  independence 
and  renewal  property,  for  any  integer  v  >  1,  we  have  En[nfn  V+1  —  /x“n  ,J  = 
rn?/As  and,  hence, 


E, 


E 

+1 


Asvf 

ai,i 

= 

mi 

TO?  Asu? 

As  m? 


Mi,„,  i 

C“n  =  EnJ2 

l—n 


did 


rm 


Thus,  C°n  reduces  to 


which  is  just  a  scaled  residual  time  term 


v?E„ 


1  - 


A X„tl  - 


m. 


where  the  ratio  is  the  time  to  the  next  arrival  divided  by  the  mean  interar¬ 
rival  time.  Thus  (A2.3)  holds.  If  the  amounts  remain  independent,  but  the 
interarrival  times  are  correlated,  then  C“n  is  just  the  residual  time  term  plus 


vt  E  E*  1 

l'Ti  l  >nAS 


'i,l+ 1 


—  T 


rrij 


and  the  condition  holds  under  broad  mixing-type  conditions  on  the  interarrival 
times.  Analogously,  if  the  independence  of  the  amounts  is  dropped,  then  the 
condition  will  hold  under  broad  mixing-type  conditions  on  the  interarrival  times 
and  amounts. 


Example.  The  simplest  example  is  where  the  <  K,  sequences  are  i.i.d., 

Ln  is  an  ergodic  Markov  chain  and  only  one  queue  can  be  selected  in  any  interval. 
Let  ASA“  denote  the  mean  number  of  arrivals  to  queue  i  in  an  interval.  Then 
C?n  =  0  in  (A2.3).  In  this  example,  let  Ui(j,x )  denote  the  probability  of 
selecting  queue  i  under  channel  state  j  and  queue  state  x.  Let  A f(j)  denote  the 
maximum  rate  of  transmission  per  unit  time  if  i  is  selected.  Then  we  have  (rate 
per  unit  time  times  As  times  the  probability  of  selecting  i)  Asgi(j,  x,  Ui(j,  x))  = 
mm{As\f(j),Xi}ui(j,x).  Suppose  that  there  are  a*(j)  with  Y^jaiU)  =  1  f°r 
all  i,  such  that  A°  —  <  0-  l-  Then  (A2.5)  holds.  This  is 

just  a  requirement  on  a  mean  flow  and  is  analogous  to  the  mean  flow  condition 
in  [14]. 


3  Proof  of  Stability 


Theorem  3.1.  Under  (A2.1)-(A2.6)  and  rule  (2.6),  the  system  is  stable. 
Proof.  For  each  i,j,n,  define 

OO 

^i,j,n  =  ~  E/  En  \l{Li  =11  _  ’ 

l=n 

W“n  -  VXi(x(nAs))Cln, 

6Vidj,n  =  ATIi(a;(nA8))pi(j,3;(rjA8),Mi(j,j;(nAs))C^nl 

where  u(-)  is  defined  in  (A2.5),  and  Cfjn  is  well  defined  by  (A2.4).  We  will 
use  the  “perturbed”  (and  time  dependent)  Liapunov  function  Vn  whose  value 
at  time  nAs  is 

=  V(x(nAs))  +  E  W?n  +  E  SVfJ>n,  (3.1) 

i  i,j 
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where  U(-)  is  defined  by  (2.3).  Let  u(-)  be  an  admissible  control  that  max¬ 
imizes  in  (2.6).  Since  Xi(nAs  +  As)  —  Xi(nAs)  =  ai<n  —  di^n  and  Endi)n  = 
As  gi(j,x(nAs),Ui(j,x(nAs)))I{Ln=j},  a  first  order  expansion  under  u(-) 
yields 


EnV(x(nA s  +  As ))  -  U(z(nAs))  =  ^  VXi(x(nAs))  [En  (ai>n  -  d^)}  +  e\ 


Enaitn  -  As  Y  gj(j,  x(nAs),Uj(j,  x(nAs)))h.Ln=ft 


+  en) 


(3.2) 


where  the  “error”  term  ei  is 


e\  =  EnV{x{nAs  +  As))  -  V(x(nAs))  -  ^  I44(;r(nAs))  [Enai>n  -  Enditn] . 

i 


By  the  fact  that  u(-)  maximizes  in  (2.6),  for  each  j, 


-  Vx‘  (x(nAS))diU’  x(nA s)>uiU’  x(nAs ))) 

<  -  'y'yVxi{x(nAs))gi(j,x(nAs),ui(j,x(nAs)).  ^  ^ 

i 

Analogously,  we  can  write 

En8V“n+ 1  -  8V»n  =  -Vxt  (x(nAs))En  [ai>n  -  A*A“]  +  e“„,  (3.4) 

where 

ei,n  =  En  [VXi{x{nAS  +  As))  -  144(z(nA-))]  C?n+1. 

Note  that  the  main  term  on  the  right  of  (3.4),  added  to  the  component  VXi  ( x(nAs))Enditn 
in  (3.2),  yields  the  “average”  VXi(x(nAs))  AsXf . 

If  Xi(nAs)  >  Kq  +  K\,  where  K\  is  the  maximum  that  can  be  transmitted 
on  an  interval  and  K0  is  defined  in  (A2.5),  then,  Xi(nAs  +  As)  >  K0  and,  by 
(A2.5),  gi(j,x(nAs),Ui(j,x(nAs)))  =  gi(j,x(nAs  +  As),Ui(j,x(nAs  +  A8))). 

Using  this,  we  have 

En8VAn+ 1  -  8Vfj  n 

=  AsI44  (x(nAs))gi(j,  x(nAs),Ui(j,  x(nAs)))  [l{Ln=j}-Uj]  +edi  j  n, 

(3.5) 

where 


ed  =  A; 

c'z,j,n 


En  [VXi(x(nAs  +  A3))  -  VXi(x(As))]gi(j,x(nAs),Ui(j,x(nAs)))C ■ 


d 


J,n+1‘ 


Note  that  the  jth  summand  in  the  brackets  in  the  right  hand  term  of  (3.2), 
with  u(-)  replaced  by  u(-)  and  added  to  the  first  term  on  the  right  of  (3.5),  is 
A.sVXi(x(nAs))gi(j,  x(nAs),  tt;(j,  x(nAs)))Uj . 
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Define  6-iti  =  —  dij.  By  the  definition  of  V (•)  and  the  binomial  expansion, 


4  =  £< 


Pi 

*£ 

1=2 


Pi 

l 


(xi(nAs)  +  hi)Pi~l  Enfi, 


i,n’ 


where  the  terms  with  the  large  parentheses  are  the  binomial  coefficients.  The 
terms  e“n  and  efj  are  bounded  by  a  constant  times  the  absolute  value  of 


Y.  En  (xi(n A3)  +  In  +  6itn)Pi  1  -  {xi(nAs)  +  hi) 


Pi- 1 


Substitute  the  inequality  (3.3)  into  (3.2).  Then  combining  the  above  computa¬ 
tions  and  canceling  terms  where  possible  yields 


EnVn+  i  —  Vn  <  0(AS)  +  en+ 


£ 


AsVXi  (a;(nAs)) 


£  9i(j,  x{nAs),Ui(j,  x{nAs)))Uj  , 


i-.XiinA^Ko+K!  I  j 

(3.6) 

where  the  0(AS)  term  is  due  to  the  components  i  for  which  Xi(nAs)  <  Kq  +  Ki, 
and  is  bounded  in  x ,  and 


en  =  e1n  +  J2  et,n  +  £  eLn  =  0(l)As  5><(nA-)  +  hi)*"2.  (3.7) 

%  i,j  i 

Note  that,  by  (A2.5),  [A“  -  9i(j,x(nAs),Ui(j,x(nAs)))Uj]  <  -c0  for  all 
i  such  that  Xi(nAs)  >  Ko.  Thus  under  any  control  u(-)  determined  by  the 
maximization  rule  (2.6),  we  must  have 

EnVn+  1-Vn<-  Y  AsVXi(x(nAs))c0  +  O{As)  +  en.  (3.8) 

i:Xi(nA.s)'>Ko~\-Ki 

The  dependence  of  e„  on  Xi(nAs)  is  dominated  by  a  constant  times  Asxp~2(nAs). 
The  main  term  on  the  right  of  (3.8)  is  of  the  order  of  '}2iXi(nAs)>K0+K1  xi~1(nA8)- 
Hence,  there  is  a  constant  AT2  such  that  if  |a;(?rAs)|  >  A"2,  then  we  can  write 

EnVn+1  -Vn<  —As  Y  VXi  (x(nAs))co/2 

i:Xi(nAs)>K0+K1  /„ 

= -As  Y  Pi  (xi(nAs)  +  /lj)Pi_1  Co/2. 

i:xi{nAa)>K0+K! 

In  particular, 

EnVnjr i  —  Vn—>  —oo,  uniformly  in  n  as  x(nAs)  — »  oo.  (3.10) 

Before  continuing,  let  us  note  the  following.  Suppose  that  x(7iAs)  were 
Markov,  and  there  were  constants  ci  >  0,A'3  >  0,  such  that  EnV(x(nAs  + 
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As)  —  V(x(nAs))  <  —ci  if  V(x(nAs ))  >  K3.  Then  the  segments  of  the  process 
V (x(nAs))  on  the  time  intervals  on  which  the  V (x(nAs))  are  no  less  than  K3  are 
supermartingales  and  classical  stability  theorems  [7]  imply  that  Ext  <  V(x)/c\ , 
where  r  is  the  return  time  to  the  set  where  V ( x )  <  K3  and  Ex  is  the  expectation 
given  initial  condition  x.  This  is  (2.2)  in  this  special  case. 

In  the  general  case  of  interest  in  the  theorem,  x(nAs)  is  not  necessarily 
Markov  and  the  Liapunov  function  is  (3.1).  The  result  (3.10)  implies  that  the 
segments  of  the  process  Vn  on  the  time  intervals  on  which  the  |x(nAs)|  are  no 
less  than  K2  are  supermartingales.  Obtaining  the  bounds  on  the  conditional 
mean  return  time  from  (3.10)  is  similar  to  what  is  done  in  the  Markov  case. 
One  just  has  to  ensure  that  the  effects  of  the  perturbations  can  be  appropriately 
“dominated.”  Using  (A2.3)  and  (A2.4),  we  have  the  bound 


E^+E^v 


0{l)YJ{xi{nAs)+hi)Pi~\ 

i 


(3.11) 


which  is  of  smaller  order  than  U(-).  Hence,  Vn  is  bounded  below  and  goes  to 
infinity  (uniformly  in  n )  as  x(nAs)  — »  00.  This  and  (3.10)  imply  the  conditions 
of  [8,  Theorem  8,  Chapter  6]  which  yields  (2. 2). 10  ■ 


4  Examples. 

The  examples  are  only  idealized  outlines  of  a  sampling  of  the  possibilities.  In 
all  cases,  we  write  x(nAs)  =  x.  Let  the  bit  interval  for  queue  i  be  be  A-1.  The 
data  arrival  process  need  not  be  specified  in  the  following  discussion. 

Example  4.1.  [Power  only  control.]  The  Shannon  capacity  of  a  white-Gaussian 
channel  with  unrestricted  alphabet  and  fixed  symbol  intervals  is  log(l  +  S'/fV)/2 
per  symbol,  where  S  and  N  are  the  signal  and  noise  powers,  resp.,  at  the  re¬ 
ceiver  [6,  Eqn.  10.17].  [For  complex  signals  multiply  by  2.]  Suppose  that  this 
form  models  all  the  channels  and  there  is  negligible  mutual  interference.  Since 
the  symbol  intervals  are  assumed  to  be  very  short  relative  to  the  scheduling 
interval,  with  appropriate  coding  the  total  number  of  packets  transmitted  over 
a  scheduling  interval  is  well  approximated  by  As  times  the  capacity  (now  mea¬ 
sured  in  packets)  of  the  channel.  The  power  constraint  is  ^2iuiU’x)  —  M  f°r 
all  j,x,  where  fi  is  the  total  available  power  and  Ui(j,x)  is  the  actual  power 
applied  to  transmission  from  queue  i  in  the  interval.  Then,  there  are  constants 
Ki ,  ctij,  Nij  such  that  if  the  channel  state  is  fixed  at  j,  then  the  channel  capacity 
Ci(j,Ui(j,x))  for  queue  i  under  Ui(j,x)  (in  packets)  is 


Ci(j,  Ui(j,  x))  =  Ki  log  [1  +  otijiiiij,  x)/Nij } .  (4.1) 

10  [8,  Theorem  8,  Chapter  6]  is  just  an  adaptation  of  the  classical  stability  results  to  cases 
where  the  perturbed  Liapunov  function  has  time  dependent  terms  that  are  dominated  by  the 
main  part  V(-)  for  large  state  values. 
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The  Ki  is  1/2  times  the  inverse  of  the  number  of  symbols  per  packet,  and  otij 
scales  for  channel  attenuation.  Then  we  can  suppose  that,  when  the  channel 
state  is  j,  di^n  =  A sgi(j,x,Ui(j,x))  =  min  {A sCi(j,Ui(j,x)),Xi}.  Due  to  the 
strict  concavity  of  the  function  log(l  +  cu),  (2.6)  might  yield  that  several  queues 
are  to  be  powered  simultaneously.  If  desired,  one  can  add  the  constraint  that 
only  one  queue  is  powered  during  any  scheduling  interval,  provided  that  (A2.5) 
holds. 

Example  4.2.  [Power  only  control.]  Now  suppose  that  the  amount  transmitted 
is  proportional  to  the  power,  and  one  might  transmit  from  several  queues  simul¬ 
taneously  with  negligible  mutual  interference.  For  each  channel  state  j,  the  rate 
of  transmission  (per  unit  time  in  the  interval)  is  assumed  to  be  proportional 
to  the  power  allocated.  This  can  be  realized  in  several  ways.  If  bandwidth  is 
the  single  resource  to  be  allocated  and  power  is  proportional  to  the  bandwidth, 
then  it  is  often  the  case  that  the  rate  is  proportional  to  power.  With  CDMA, 
one  possibility  is  to  allow  several  “spreading  sequences”  per  queue  [15]. 

Another  possibility  is  to  allow  the  bit  interval  (duration)  to  be  controllable. 
In  particular  suppose  that  (for  queue  i)  there  are  basic  time  intervals  of 
which  the  actual  bit  interval  A/  is  one  of  the  multiples  m8\,m  =  l,...,rn,, 
for  some  given  ?n,;  <  oo.  Suppose  that  the  channels  are  white-Gaussian,  that 
there  is  perfect  synchronization  at  the  receiver,  and  that  the  system  design  is 
such  that  if  the  power  allocation  to  any  queue  is  not  zero  then  it  must  be  such 
that  the  SNR  per  bit  at  the  receiver  is  at  least  some  given  positive  number 
Ci(j).  This  minimal  amount  Cj(j’),  together  with  the  bit  interval,  determines 
the  capacity.  Thus,  cutting  the  bit  interval  in  half  would  require  twice  the 
power  for  the  same  SNR.  The  chosen  bit  interval  is  constant  during  a  scheduling 
interval.  Summarizing,  we  suppose  that  there  are  constants  A such  that  the 
number  of  packets  per  unit  time  that  can  be  sent  from  queue  i  under  channel 
state  j  is  A f(j)ui(j,x)  for  the  finite  number  of  allowed  values  of  Ui(j,  x).  Also 
A sgi{j,x,Ui(j,x))  =  . 

Example  4.3.  [Control  power  and  time.]  In  Examples  4.1  and  4.2,  the  total 
power  was  constrained  to  p  at  each  time  and  the  transmissions  from  all  selected 
queues  in  an  interval  were  done  simultaneously.  Alternatively,  suppose  that  we 
schedule  by  dividing  time  as  well  as  power  in  the  following  generalized  form  of 
TDMA.  For  each  queue  i  and  channel  and  queue  state  vector,  partition  the  nth 
scheduling  interval  into  K  subintervals  (some  subintervals  might  have  length 
zero)  of  total  lengths  A f(j,x),i  <  K ,  where  ]>A  Af  (j,  x)  =  As.  Summarizing, 
in  the  nth  scheduling  interval,  transmit  from  queue  i  on  a  total  time  A|n,  which 
can  depend  on  the  queue  i  and  on  the  current  queue  and  channel  states  x,j. 

Since  the  scheduling  intervals  are  short,  we  might  be  allowed  to  vary  the 
power  within  the  interval,  provided  that  it  averages  to  p  over  the  interval.  In 
particular,  let  Pi(j,x)  denote  the  power  applied  to  queue  i  during  the  part  of 
the  interval  that  queue  i  is  being  worked  on.  Then  the  full  vector-valued  control 
Ui(j,  x)  is  (pi(j,x),Af(j,x)).  Then,  in  lieu  of  the  pointwise  constraint  (2.1),  we 
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now  constrain  average  total  power  over  the  interval  and  (2.1)  is  replaced  by 

Y^Pi(j,x)[Asi(j,x)/As]<  p,,  all  j.  (4.2) 

i 

Consider  the  setup  in  Example  4.1,  but  where  only  one  queue  can  be  powered 
at  a  time  and  the  scheduling  interval  is  divided  among  the  queues,  as  above. 
Then,  the  rule  (2.6)  is  the  maximizing  (power  and  time)  value  in 

max  {e  VXi  (x)Af  ( j ,  x)Ki  log  [1  +  aijPi(j,  x)/Nij ]  j  , 

with  constraints  (4.2)  and  =  A s .  Then  A sgi(j,x,Ui(j,x))  = 

min  {A sKt  log  [1  +  a^p.^j,  x)/Nij j ,  zj  . 

Example  4.4.  The  general  scheme  is  not  restricted  to  systems  with  only  one 
antenna,  and  this  will  be  briefly  illustrated.  Suppose  that  there  are  two  an¬ 
tennas  with  total  power  constraints  ua,a  =  1,2.  First  allow  the  two  antennas 
to  be  scheduled  independently.  One  simply  uses  (2.6)  for  each  antenna.  Let 
ja  denote  the  channel  state  from  antenna  a  =  1,2.  When  the  antennas  are 
scheduled  independently,  then  there  are  two  controls  uf  ( ja ,  x)  and  functions 
gi‘{ja,x,uf(ja,x)),  a  =  1,2.  The  two  antennas  might  be  used  for  the  same 
or  for  different  queues.  Alternatively,  one  might  coordinate  the  use  of  the  an¬ 
tennas,  for  example,  by  using  space  time  coding  [13].  This  gives  a  different 
function  <?(•).  More  generally,  one  would  compare  a  set  of  reasonable  control 
alternatives  for  each  channel  state,  so  that  the  best  use  of  the  resource  is  ob¬ 
tained.  Some  queues  might  be  scheduled  on  only  one  antenna  and  others  on 
both,  either  independently,  or  via  (for  example)  space-time  coding. 


5  Examples  of  Controls  and  Relation  to  Optimal 
Controls 

Consider  Example  4.2,  where  the  rate  of  transmission  is  proportional  to  power 
(the  model  used  in  [1])  and  let  V(x)  =  —  1-  Then  the  control  given 

by  (2.6)  is  the  maximizer  in 

max  <|  As  E  x) 

over  the  allowed  discrete  values.  In  the  rest  of  the  discussion  of  the  controls,  we 
ignore  the  small  queues,  which  are  readily  accounted  for. 

A  two-dimensional  case.  Consider  a  two-queue  and  two-channel  state  ex¬ 
ample  with  no  constraints  on  the  power  allocation.  Let  Af(l)  >  A^l),  and 
Af(2)  <  A^(2).  For  channel  state  1,  apply  all  power  to  queue  1  when 

x,/x2>  [A^(l)/A?(l)]1/P.  (5.1) 
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For  channel  state  2,  apply  all  power  to  queue  2  when 

x2/x1>[Xd1(2)/Xd2(2)]1/p .  (5.2) 

As  p  — >  oo,  the  switching  lines  move  to  the  diagonal. 

An  alternative  Liapunov  function.  Now  consider  the  Liapunov  function 
V(x)  =  'Yji[xi  +  hi]p+1.  With  equal  hi,  this  gives  results  that  are  closer  to  the 
numerically  computed  optimal  controls  for  an  approximating  diffusion  with  cost 
rate  max-faq,^}  (see  below).  The  inequalities  (5.1)  and  (5.2)  are  replaced  by. 
respectively, 


x\  +  hi/x2  +  hi  >  [Ag(l)/A?(l)]  ^  , 

(5.3) 

*2  +  h2/xi  +  hx  >  [A^  (2) 7X3(2)] 1,P  . 

(5.4) 

Here  the  cost  rate  is  less  sensitive  to  differences  in  the  xt  unless  they  are  large. 

Comparison  with  optimal  controls.  It  is  interesting  to  relate  the  control 
forms  determined  by  ((5.1),  (5.2))  and  by  ((5.3),  (5.4))  to  optimal  controls 
for  diffusion  processes  with  similar  dynamics.  The  following  figure  gives  the 
optimal  switching  surfaces  for  control  problem  for  the  diffusion  process  dx  = 
g(x,u(x))dt  +  crdw.  The  qualitative  form  of  the  controls  did  not  depend  on  a. 
We  let  Af(l)  >  A^l),  Xf  (2)  <  A!f(2),  and  the  optimal  controls  are  obtained  by 
numerical  solution  using  the  methods  of  [9].  The  cost  rate  is  k(x)  =  maxjxi,  £2} 
for  the  left-hand  figure  and  JT  x,  for  the  right-hand  figure.  The  total  cost  was 
either  of  the  ergodic  or  discounted  form  with  small  discount  factor,  with  little 
difference  in  the  controls  between  them.  Similar  results  hold  for  discrete  time 
problems.  The  slopes  for  the  linear  cost  rate  tend  to  be  larger  than  those  for 
the  maxjj)!,^}  criterion.  Equivalently,  for  the  linear  cost  rate  we  put  more 
emphasis  on  using  the  most  efficient  channel  and  less  on  attaining  closeness  of 
the  queues.  Note  the  similarity  of  the  controls  to  the  forms  given  by  ((5.1), 
(5.2))  and  ((5.3),  (5.4)).  The  controls  in  the  left  hand  figure  have  the  form  of 
((5.3),  (5.4)),  and  those  in  the  right  hand  figure  have  the  form  of  ((5.1),  (5.2)). 
Thus,  with  appropriate  choices  of  the  parameters,  the  controls  are  close  to  those 
for  optimal  processes  for  closely  related  problems. 
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Figure  5.1.  Switching  curves  for  the  optimal  control: 
k(x)  =  max{xi,a:2}  and 
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